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ABSTRACT 

E-learning systems, or learning management systems, as it is known in the field, sit at the heart of educational systems 
and are used to systematically deliver on-line content and facilitate the learning experience around that content. It 
becomes essential to ensure that Learning Management Systems of a high standard are being developed. In the field of 
Systems Development Methodologies (SDMs), much research has been done and the value of SDMs is proven and 
documented. To ensure that e-leaming systems of outstanding quality are being developed, it is crucial that systems 
development methodologies are being used as they can have a significant impact on the development process. 

The main focus of this study was to determine if systems development methodologies are being utilised in the 
development of electronic learning systems in South Africa and if these methodologies are being applied effectively. By 
utilising a survey as the main research method and statistically analysing the dataset, meaningful results were obtained. 
Some of the findings were that academic institutions are clearly using open-source LMSs more than proprietary LMSs. 
The LMS platform had a moderate effect on the success of the LMS. Open-source LMS users were very satisfied with 
their LMS platform and not inclined to consider a proprietary LMS for future projects. Proprietary LMS users may be 
inclined to consider an open-source LMS for future projects. It was found that a very strong relationship exists between 
respondents that indicated that they use RAD as SDM and the impact this SDM had on the quality and productivity of the 
development process. It was found that a very strong relationship exists between the perceived support that the SDM 
provides and the impact of the SDM on the quality and productivity of the development process. This in turn has a strong 
influence on the impact systems development methodologies have on the quality of learning management systems. 
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1. INTRODUCTION AND PROBLEM STATEMENT 

With the dramatic increase in the popularity and use of e-learning and e-learning systems, it becomes a 
necessity to ensure that systems of a high quality are being developed. If e-learning is applied efficiently, it 
can go a long way towards helping learners to generate and obtain knowledge for themselves. Many cultures 
around the world still have no access to even basic education. E-learning can be one of the answers to this 
problem. 

Even a cursory study into e-learning makes it clear that there is still plenty of confusion surrounding e- 
learning and the technology and applications involved with it. It becomes apparent there is still much to 
benefit, from further research into this field. 

In another and not unrelated field of Systems Development Methodologies (SDMs), much more research 
has been done and the value of SDMs is proven and documented. However, there is still a lack of empirical 
evidence on the actual use and effectiveness of SDMs and even more so with its use and effectiveness in the 
development of e-learning systems. 

SDMs have had a major impact on the development of software systems over the last 40 odd years and 
are an indispensable tool for developing systems in recent times. E-learning systems, being an emerging 
subset in software systems technology, could well profit from the benefits SDMs have to offer. 

The aim of this study will be to research the use and effectiveness of SDMs in the development of e- 
learning systems. 
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2. LITERATURE STUDY 

2.1 Terminology 

For the purposes of this study e-learning can be seen as a medium for delivering and facilitating learning, 
through electronic means. 

The term Learning Management System will also be used for the purposes of this study. An LMS is a 
software system that is used to systematically deliver content on-line and to facilitate the learning experience 
around that content. 

The following definition of a systems development methodology will be used (Huisman, 1999): 

• Systems development approach 

This can be defined as the philosophical view on which the methodology is built. Thus, the set of 
goals, guiding principles and beliefs, basic concepts and principles of the systems development 
process that drive interpretations and actions in systems development (Iivari et al., 1998; Iivari el 
al., 1999). Examples of systems development approaches are the structured approach, object- 
oriented approach, information modelling, etc. 

• Systems development process model 

Wynekoop and Russo (1993) define a process model as a representation of the sequences of 
stages through which a system evolves. Some examples of process models are the linear life-cycle 
model, the spiral model and incremental model. 

• Systems development method 

A method is a systematic approach to conducting at least one complete phase of systems 
development, consisting of a set of guidelines, activities, techniques and tools, based on a particular 
philosophy of systems development and the target system (Wynekoop & Russo, 1993). Examples 
include IE, SSADM, etc. 

• Systems development technique 

A systems development technique consists of a well-defined sequence of actions, ensuring 
successful results if used correctly (Iivari et al., 2000; Brinkkemper, 1996), for example entity 
relationship diagrams and data flow diagrams.. 

2.2 The Use of Systems Development Methodologies 

It is argued that the use of SDMs will aid in the development process by making it more effective, secure, 
predictable and easier to control, whilst improving productivity and quality (Fitzgerald et al., 2002). Whether 
systems development methodologies are being used in practice has been a point of interest for academics for 
an extended period of time. 

Russo et al. (1995) found that only 6% of developers followed an SDM rigorously. Chatzglou and 
Macaully (1996) found that 47% of their population had never used a methodology. Avison and Fitzgerald 
(2003) also found this number to be low. Less than 50% of organisations follow SDMs strictly (Glass, 1999). 
Huisman and Iivari (2006) determined that when SDMs are being used in practice, it is not to the full extent. 
Fitzgerald et al. (2002) show that the use of formalised SDMs was significantly higher in larger organisations 
(more than 1 000 employees) and larger IS departments (more than 20 personnel). In recent studies 
conducted by Brits (2011) and Janse van Rensburg (2012) it was found that 74% of organisations in South 
Africa make use of SDMs. Conradie (2010) and Wagener (2012) also found this number to be above 75%. 

2.3 The Effectiveness of Systems Development Methodologies 

Empirical research on the effectiveness of SDMs is very limited, which seems to be due to the lack of 
standard criteria that measures SDM effectiveness (Conradie 2010). This section will confer the results on the 
effectiveness of SDMs that were found in the literature. 

Although many SDMs have been successfully utilised over the years, there have been many software 
failures, which has questioned the relevance of SDMs. Even with the use of SDMs, projects are still being 
abandoned halfway, still overshooting the budget, and still not being delivered within an appropriate time 
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frame (Truex et al., 2000; Gruner et al., 2007). However, great strides have been made in the last four 
decades; yet there is still much to be learned from studies into the effectiveness of SDMs. 

There are several compelling reasons for using SDMs. Fitzgerald et al. (2002) state that systems 
development methodologies have been endorsed by the literature, as being capable of ensuing that the 
development process is more effective, secure, predictable and easier to control. Gruner et al. (2007) argue 
that it is challenging and precarious to develop software without the guidance or structured process that an 
SDM can provide. Huisman and Iiavari (2006) reasoned that SDMs introduce structure to the design process, 
thereby improving the effectiveness of software development and thus ensuing more consistent outcomes. 
The use of software methodologies is said to decrease the risk of failure of an information system (IS) project 
(Hull et al, 2002; Avison & Fitzgerald, 2003). 


3. DATA COLLECTION AND ANALYSIS 

In the field of information systems, surveys are a popular strategy to employ in the collection of empirical 
evidence. A questionnaire, as the measurement element, was developed in collaboration with a statistical 
consultation service. The constructs, identified from the conceptual research model, were operationalised by 
selecting measurement scale items (questions) and scale types. The questions were adapted from previous 
research studies, which proved to be reliable. The questionnaire was concise and relevant and contained 
mostly leading, importance and 5-point Likert scale questions. It was distributed electronically, as a macro- 
enabled Excel file, to personnel at institutions of higher education in South Africa that are responsible for 
developing and/or deploying e-learning systems. Software companies in South Africa, which develop e- 
learning solutions, were also targeted. The questionnaire had extensive built-in entry validation to ensure that 
the respondents fill in the correct values as well as coding to assist in data analysis. 

Fifty responses were received from a possible one hundred and twenty-five responses. Therefore, the 
participation rate equalled 40% with 50 cases available for data analysis. 


4. RESULTS OF THE RESEARH QUESTIONS 

The relationship between industry and platform of the LMS (RQ1): 

In academic institutions open-source LMSs are being used in almost 61% of the cases and in the private 
sector just above 52%. Many of the academic institutions indicated that they are in the process of moving 
towards an open-source learning management system but that they are currently still using a proprietary 
LMS. 

The relationship between industry and number of students (RQ2): 

Crosstab analysis was done to determine if the type of industry relates to the number of users of the 
specific LMS in question. The results of the crosstab analysis can be seen in Table 1 and it was statistically 
significant with p < 0.01 and a Cramer’s V of 0.568. 

It is quite noticeable that 75% of the academic institutions’ LMSs are being utilised for 10 000 or more 
students, learners or employees. The private sector almost exclusively uses LMSs with more than a 1 000 
users. 


Table 1 . Crosstab analysis: Research Question 2 


Industry 

1-99 

100-499 

Number of students/learners/employees using the LMS 
500-999 1000-4999 5000- 9999 10 000 or more 

Total 

Academic 

7.1% 

10.7% 

0% 

0% 

7.1% 

75% 

100.0% 

Private 

Sector 

0% 

15.8% 

0% 

21.1% 

31.6% 

31.6% 

100.0% 

Total 

4.3% 

12.8% 

0% 

8.5% 

17% 

57.4% 

100.0% 
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The relationship between industry and method of procurement of LMS (RQ3): 

Crosstab analysis was to determine if the type of industry relates to the LMS platform being used. The 
results of the crosstab analysis can be seen in Table 2 and it was statistically significant with p < 0.05 and a 
Cramer’s V of 0.447. There is very little in-house development being done when it comes to the procurement 
of LMSs with a combined (academic and private sector) total of 17.4% of the respondents indicating that 
they make use of in-house development. With 32.6% of the procurement of LMSs being done by adapting 
open-source LMSs, it is by far the preferred method of procuring an LMS. To elaborate on the 
abovementioned adaption of open-source LMS, it is worth noting that 39.3% of academic institutions use 
open-source systems “as is”, while only 5.6% of the private sector use LMSs “as is”. Only 21.4% of 
academic institutions adapt their LMSs compared to the 50% of the private sector. A combined total of 
23.9% of the procurement of LMSs are being done by purchasing off the shelf products. 


Table 2. Crosstab analysis: Research Question 3 





Procurement of LMSs 


Industry 

Purchased 

In-house 

Open-source 
(used as-is) 

Adapted from open- 
source system 

Total 

Academic 

25% 

14.3% 

39.3% 

21.4% 

100.0% 

Private 

Sector 

22.2% 

22.2% 

5.6% 

50% 

100.0% 

Total 

23.9% 

17.4% 

26.1% 

32.6% 

100.0% 


The relationship between the LMS platform and perceived success of the LMS (RQ4): 


T-tests indicated that the LMS platform had a medium effect on the success of the LMS with an effect 
size [f ~ ) of 0.34 and a statistical significance, p > 0.05. 

The relationship between the LMS platform and the satisfaction with the platform (RQ5): 

T-tests indicated that the platform of the LMS has a very high effect on the inclination of respondents to 
consider using an open-source LMS for future projects with an effect size (L) of 1.7 1 and it is statistically 
significant with p < 0.001. Respondents that use an open-source LMS would be inclined to keep using an 
open-source LMS for future projects with a mean value of 4.4. 1 Respondents that use a proprietary LMS will 
be less inclined to consider an open-source LMS for future projects with a mean value of 2.75. T-tests also 
indicated that the platform of the LMS has a very high effect on the inclination of respondents to consider 

using a proprietary LMS for future projects with an effect size (f 2 ) of 1.04 and it is statistically significant 
with p < 0.001. Respondents that use a proprietary LMS would be inclined to keep using a proprietary LMS 
for future projects with a mean value of 3.65. Respondents that use an open-source LMS will be less inclined 
to consider a proprietary LMS for future projects with a mean value of 2.4. 

The relationship between the procurement method of the LMS and the satisfaction with 
the platform (RQ6): 

T-tests indicated that the procurement method of the LMS has a very high effect on the inclination of 
respondents to consider using an open-source LMS for future projects. Respondents that use an open-source 
LMS would be inclined to consider using an open-source LMS (as is, i.e. not adapting it) for future projects 

with an effect size (/ ) of 2.16 and it is statistically significant with p < 0.001. Respondents that use a 
proprietary LMS would be inclined to consider using an open-source LMS (as is, i.e. not adapting it) for 
future projects with an effect size (/~) of 2.11 and it is statistically significant with p < 0.001. Respondents 
that use an open-source LMS would also be inclined to consider adapting an open-source LMS for future 
projects with an effect size (/ ) of 1.50 and it is statistically significant with p < 0.001. Respondents that 


1 All the means are indicated as a value out of a possible 5, as a five-point Likert scale was used. 
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use a proprietary LMS would also be inclined to consider adapting an open-source LMS for future projects 
with an effect size (/ ' ) of 1.45 and it is statistically significant with p < 0.001. 

It is clear that the respondents will consider using open-source LMSs for future projects, but they are less 
inclined to adapt such LMSs, than using them as is. 

The relationship between the perceived success of the LMS and the satisfaction of the 
platform (RQ7): 

There is a strong relationship that exists between the success of the LMS and the satisfaction of the 
platform used with a Spearman’s rho of 0.589 and it is statistically significant with p < 0.001. 


The relationship between the type of SDM used and the success of the LMS (RQ8): 

T-tests indicated that the use of formal SDMs had a medium effect on the success of the LMS with an 
effect size (/' ) of 0.34 and a statistical significance, p < 0.05. 


The relationship between the type of SDM used and the impact of the SDM on the 
development process (RQ9): 

This research question determines if there is a relationship between the type of formal standard SDM used 
and the impact of the SDM on the quality and productivity of the development process. There is a very strong 
relationship that exists between respondents that indicated that they use RAD as SDM and the impact this 
SDM had on the quality and productivity of the development process. Spearman’s rho = 0.618. p < 0.001. 
There is a strong relationship that exists between respondents that indicated that they use IE as SDM and the 
impact this SDM had on the quality and productivity of the development process. Spearman’s rho = 0.517. p 
< 0.01. The SDM with the strongest influence on the quality and productivity of the development process is 
RAD, which is an Agile methodology. 

The relationship between the performance expectancy of the SDM and the impact of the 
SDM on the quality and productivity of the development process (RQIO): 

There is a strong relationship that exists between the performance expectancy of the SDM and the impact 
of the SDM on the quality and productivity of the development process with a Spearman’s rho of 0.563 and it 
is statistically significant with p < 0.01. 

The relationship between the performance expectancy of the SDM and the perceived 
impact of the SDM on the quality of the LMS (RQll): 

There is a strong relationship that exists between the performance expectancy of the SDM and the 
perceived impact of the SDM on the quality of the LMS with a Spearman’s rho of 0.547 and it is statistically 
significant with p < 0.01. 

The relationship between the perceived support that the SDM provides and the impact of 
the SDM on the quality and productivity of the development process (RQ12): 

There is a very strong relationship that exists between the perceived support that the SDM provides and 
the impact of the SDM on the quality and productivity of the development process with a Spearman’s rho of 
0.744 and it is statistically significant with p < 0.001. 

The relationship between the perceived support that the SDM provides and the perceived 
impact of the SDM on the quality of the LMS (RQ13): 

There is a strong relationship that exists between the perceived support that the SDM provides and the 
perceived impact of the SDM on the quality of the LMS with a Spearman’s rho of 0.541 and it is statistically 
significant with p < 0.01. 
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The relationship between the impact of the SDM on the quality and productivity of the 
development process and the success of the LMS (RQ14): 

There is a weak relationship that exists between the impact of the SDM on the quality and productivity of 
the development process and the success of the LMS with a Spearman’s rho of 0.177 and it is not statistically 
significant with p > 0.05. 

The relationship between the impact of the SDM on the quality and productivity of the 
development process and the perceived impact of the SDM on the quality of the LMS 
(RQ15): 

There is a strong relationship that exists between the impact of the SDM on the quality and productivity 
of the development process and the perceived impact of the SDM on the quality of the LMS with a 
Spearman’s rho of 0.543 and it is statistically significant with p < 0.01. 

The relationship between the perceived impact of the SDM on the quality of the LMS and 
the success of the LMS (RQ16): 

There is a moderate relationship that exists between the perceived impact of the SDM on the quality of 
the LMS and the success of the LMS with a Spearman’s rho of 0.376 and it is statistically significant with p 
<0.05. 

The need to design an SDM specific to LMS and the SDM that was used to design the 
current LMS (RQ17): 

There is a strong negative relationship that exists between respondents that indicated that they use another 
(not listed) SDM and on opinion that there is room for a newly designed SDM specifically for developing 
LMSs with a Spearman’s rho = -0.402 and a statistical significance, p < 0.05. There is a strong relationship 
that exists between respondents that indicated that they use another (not listed) SDM and on the opinion that 
an existing SDM can be adequately adopted to suit the needs of e-learning with a Spearman’s rho = 0.437 
and a statistical significance, p < 0.01. There is a moderate negative relationship that exists between the 
respondent that indicated that they use IE (Information Engineering) as SDM and the opinion that an existing 
SDM can be adequately adopted to suit the needs of e-learning with a Spearman’s rho = -0.399 and a 
statistical significance, p < 0.05. 

Summary of the results 

A summary of the statistical analyses that were performed, and the relationships that were evaluated, can 
be seen in table 3. 


Table 3. Research questions as tested (Note: p = Level of Significance. ***=<0.001, **=<0.01, *= <0.05, ‘=<0.1; 

n/a=this type of test could not/was not performed) 


Research 

Question 

Path tested 

Spearman’s rho 
(p) / Cramer’s 

vO) 

Effect Size 

(L) 

Relationship / 
Correlation / Level 
of Association 

RQ1 

Industry — > Platform Used 

(p c = 0.709*** 

n/a 

Very strong 

RQ2 

Industry — > Number Users 

cp c = 0.568** 

n/a 

Very strong 

RQ3 

Industry — * Procurement LMS 

(p c = 0.447* 

n/a 

Very strong 

RQ4 

Platform_Used — » Success LMS 

n/a 

0.34 

Medium effect 

RQ5 

Platform_Used — > 

n/a 

j 72*** 

Very high effect 


Satisfaction_LMS_Platform 


2 04*** 


RQ 6 

Procurement LMS — » 

n/a 

2 22 *** 

Very high effect 


Satisfaction_LMS_Platform 


2.16*** 





1.50*** 





2 45 *** 


RQ7 

Success LMS — > 

p = 0.589*** 

n/a 

Strong 


Satisfaction_LMS_Platform 




RQ 8 

Fonnal SDM Use — > Success LMS 

n/a 

0.34* 

Moderate 
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Research 

Question 

Path tested 

Spearman’s rho 
(p) / Cramer’s 

vO) 

Effect Size 

(D 

Relationship / 
Correlation / Level 
of Association 

RQ9 

Extent_Standard_SDM (RAD) — > 
Impact_SDM_Quality_Productivity 

p =0.618*** 

n/a 

Very strong 

RQ10 

Performance Expectancy SDM — » 
Impact_SDM_Quality_Productivity 

p = 0.563** 

n/a 

Strong 

RQ11 

Perfonnance Expectancy SDM — * 
Impact_Quality_LMS 

p =0.547** 

n/a 

Strong 

RQ12 

Perceived Support SDM — > 
Impact_SDM_Quality_Productivity 

p = 0.744*** 

n/a 

Very strong 

RQ13 

Perceived Support SDM — > 
Impact_Quality_LMS 

p = 0.541** 

n/a 

Very strong 

RQ14 

Impact SDM Quality Productivity — * 
Success LMS 

p = 0.177’ 

n/a 

Weak 

RQ15 

Impact SDM Quality Productivity — » 
Impact_Quality_LMS 

p =0.543** 

n/a 

Strong 

RQ16 

Impact_Quality_LMS — » Success LMS 

p = 0.376* 

n/a 

Moderate 

RQ17 

Need_SDM_LMS -» 
Extent_Standard_SDM (Other) 

p =0.437** 

n/a 

Strong 


5. CONCLUSION 

The inferential statistics that are presented in this paper reveal some interesting facts about the relationships 
between certain constructs related to learning management systems and systems development methodologies 
in South Africa. 

Crosstab analysis indicated that open-source learning management systems are preferred to proprietary 
LMSs, especially in the academic sector. It also revealed that the majority of LMSs in South Africa are being 
used for 10 000 or more users. Academic institutions also tend to use open-source LMSs as is and the private 
sector adapts open-source LMSs to a large extent. 

There is a strong relationship that exists between the perceived success of the LMS and the satisfaction of 
the platform used. It makes sense that respondents will be satisfied with their LMS platform if the LMS is 
successful. In general, respondents are very satisfied with their current LMS platform. 

Open-source LMS users are not inclined to consider a proprietary LMS for future projects. Proprietary 
LMS users are more inclined to consider an open-source LMS for future projects. Both proprietary and open- 
source LMS users will be less inclined to adapt an open-source LMS, than using it as is. 

A very strong relationship were found between the perceived support that the SDM provides and the 
impact of the SDM on the quality and productivity of the development process. There is a very strong 
relationship that exists between respondents that indicated that they use RAD as SDM and the impact this 
SDM had on the quality and productivity of the development process. This correlates with what the literature 
revealed on the use of Agile methodologies in the post-methodology era. 

There is also a strong relationship that exists between the performance that users expect SDMs can 
provide and the impact that the specific SDMs have on the quality and productivity of the development 
process. A very strong relationship exists between the support that the SDM provides in the development 
process and the actual impact that the SDM has on the quality and productivity of the development process. 

Both the perceived support that the SDM provides and the impact of the SDM on the quality and 
productivity of the development process have a strong influence on the impact the SDM has on the quality of 
the LMS. 

The study made a contribution to the discipline of information systems, and more specifically, learning 
management systems, by providing insights with regard to the factors effecting the use and effectiveness of 
SDMs in developing LMSs. As far as could be ascertained, this study generated the first empirical data on the 
procurement and development of LMSs in South Africa. A holistic picture was drawn on what the South 
African e-learning market looks like and it was determined that the extent of use of open-source LMSs 
exceeds what was initially believed. 
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Almost 75% of the respondents indicated that there may be room for a newly designed SDM specifically 
for the development of LMSs and tools for LMSs. This could well be a great opportunity for future research 
in this field. This study may also serve as a stimulus for future research in the field of learning management 
systems and more specifically the development of LMSs by using systems development methodologies to 
enhance the chances of success for those systems. 
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